Graphical user interface programs (GUI) are known in the art which provide an interactive display allowing a user to display different images on overlapping or different portions of a video display. Examples of such graphical user interfaces are the WINDOWS.TM. program or the IBM.TM. OS/2.TM. program. Other graphical user interface programs, sometimes referred to as graphical window operating software, are also known in the art.
Such graphical user interfaces are particularly useful in so-called multimedia applications where different types of media are combined to present information to a user. For example, graphics, text and audio may be incorporated into an informational presentation. In addition, segments or portions of live action, real time, or full motion video may also be incorporated into such a presentation.
For the purposes of this application, the term "motion video" is interpreted to mean any video segment or presentation including live action, real time, or full motion video. Examples of motion video include, but are not limited to, NTSC, PAL or SECAM type television signals, or the like, including live television signals or broadcasts, cable television signals or the like, or motion picture video, suitably digitized and converted into a format suitable for presentation on a computer display. The term motion video may also include, but is not limited to any computer generated display or display segment, including computer animation or the like, CD-ROM playback, or video recorded on a VCR or other recording means.
Several problems arise when trying to integrate motion video into a graphical user interface environment. Usually the integrated motion video is displayed by placing a window somewhere within the graphics display screen. This window within the graphics display screen where the motion video is contained may be referred to as a motion video window (MVW) or motion video display window. In typical prior art video controllers, such as a VGA video controller, a graphics image (graphics mode) or ASCII text (text mode) is displayed on a screen (e.g., flat panel display, CRT or the like) having a particular pixel resolution and pixel depth. Pixel resolution refers to the number of pixels which are used to constitute the display, for example, 640.times.480 pixels. Other resolutions are also possible, such as 800.times.600, 1024.times.768, or the like. Pixel depth refers to the number of bits used to represent each pixel. The number of bits used for each pixel is indicative of the number of colors or gray scales that each pixel may represent. For example, a pixel depth of 4 bits per pixel provides 24 or 16 gray scale levels or colors. VGA or SVGA controllers may support various pixel depths such as 4, 8, 16, or even 24 bits per pixel.
For motion video, it has been determined empirically that in order to present a realistic motion video display, a reasonably large pixel depth is required. For example, a pixel depth of at least 16 bits per pixel may be necessary in order to provide a realistic motion video display having a range of colors or grey scales which appear realistic to the viewer.
For displays having a low pixel resolution, such a pixel depth may not be unduly difficult to achieve, if the length of the video segment or number of frames per second is limited. For example, a 640.times.480 display contains 307,200 pixels. FIG. 1A shows a simplified block diagram of a typical prior art VGA graphics controller, graphics memory and video displays. In a system shown in FIG. 1A, graphics memory 101 may comprise a random access memory 32 bits wide. Assuming a pixel depth of sixteen bits, for each fetch from graphics memory 101, video graphics controller 102 may retrieve two sixteen-bit pixel words. Typical video displays such as TV 103 have a 30 Hz interlaced refresh rate, while CRT 104 or flat panel display 105 may have refresh rates in the range of 60 to 75 frames per second. Assuming a refresh rate of 60 Hz and a pixel resolution of 640.times.480, SVGA graphics controller 102 would need to retrieve more than 18 million pixel words per second from graphics memory 101. Graphics memory 101 is 32 bits wide, and two sixteen bit pixel words are retrieved with each fetch from graphics memory 101. Thus, a total of 108.5 nanoseconds may be required for each fetch, a speed within the range of memories available in the prior art.
However, graphics memory 101 must also be written to periodically by the host computer (not shown) in order to provide image data to be displayed. Assuming graphics controller 102 makes eight fetches from graphics memory 101 for every one CPU access to graphics memory, the amount of time needed for each fetch would be reduced to 96.5 nanoseconds, a figure still within range of commercially available memories.
However, if the video resolution is increased, for example to a 1024.times.768 display, a fetch rate of 37.7 nanoseconds per fetch would be required. In order to support such a display, a much faster graphics memory would be required such as more expensive SRAM, dual port RAM, or other high performance expensive DRAM organized as 64 bit wide data path. As refresh rates increase, for example to 75 Hz and as pixel depths increase, for example to 24 bits per pixel, a point may be reached where the fetch rate needed to supply the video display is higher than that from commercially available memories.
One technique for limiting memory bandwidth requirements is to reduce the number of frames per second of the motion video (e.g., 15 frames per second). This solution may reduce the amount of graphics memory necessary and decrease the required graphics memory bandwidth. However, limiting the number of frames per second may give the video a jerky stop-motion effect, especially when used for slow-motion playback.
Prior solutions of providing a motion video window on a graphics display used extra hardware components and overlayed the motion video window on top of the graphics window such that the graphics window operating software may be ignorant of the motion video display window. Thus one could not easily manage the motion video window such as moving it from location to location on the screen or resizing the motion video window using the graphics window operating software. It is desirable to further integrate the motion video window with the graphics window operating software such that the motion video window can be readily managed on the display screen.
In addition, another problem occurs when attempting to display motion video in a graphical user interface when running a graphics window operating software such as Microsoft Windows. Many multimedia resources, for example encyclopedias or other reference materials packaged into a CD-ROM format, may incorporate segments or portions of motion video recorded at a different pixel depths. In order to display these motion video sequences, the graphical user interface (GUI) program must operate at the same pixel depth as the recorded motion video or be capable of supporting multiple pixel depths using prior art super VGA controllers. If the graphical user interface program is operating at a first pixel depth (e.g., 8 bits per pixel) and the motion video is recorded on a CD-ROM at a different pixel depth (e.g., 16 bits per pixel), either an error message may be generated by the GUI and the video segment will not be displayed or the CD-ROM will be displayed at the pixel depth of the graphics, so the playback will not use the higher pixel depth available on the CD-ROM. If the error message occurs, the user must then reconfigure his graphical user interface program to the same pixel depth (i.e., change video display mode) as that used to record the motion video on the CD-ROM. Such reconfiguring steps may not be user-friendly and present an additional difficulty in providing the multimedia presentation. In addition, different multimedia resources may use different pixel depths for recording motion video, and thus the pixel depth for the graphical user interface program must be reset prior to displaying different video segments. In conclusion in order to playback a CD-ROM or other motion video at its highest pixel depth the user will normally need to restart windows for that highest pixel depth and then run the CD-ROM or other motion video application.
A prior art method of combining graphics and video onto a single display is illustrated by FIG. 1B. In this case the video signal is "overlayed" on top of the graphics image by multiplexing video pixel data and graphics pixel data with a VGA graphics controller. A composite video signal 130 is generated by a composite video generator 110 such as a TV camera or home video camera or other composite video source. The composite video signal 130 is decoded into a YUV signal 132 by a video decoder 112 such as an NTSC/PAL decoder, MPEG CODEC, or other decoder. The YUV signal 132 is fed into the digital video processor 114 to generate the video pixel data 138. An exemplary digital video processor 114 is a CL-PX2070 manufactured by Pixel Semiconductor. The digital video processor 114 requires a video frame buffer memory 116 communicating address and data over bus 134. Control commands are communicated from the CPU 118 onto the system bus (VL, Local, or PCI Bus) 136 into the bus controller 113 which passes the commands to the digital video processor 114 via the ISA bus 135. VGA graphics controller 128 is coupled to the digital video processor 114 via video pixel data bus 138. Bus 138 is referred to as a feature connector. The VGA graphics controller 128 requires a graphics memory 120 to store graphic data communicating address and data over bus 140. The simplified block diagram of the VGA graphics controller 128 contains a RAM DAC 126, multiplexer 124, VGA control and pixel data generation logic 122. Graphics display information from the CPU 118 is driven onto the system bus (VL, Local, or PCI Bus) 136 received by the control logic 122 and stored in graphics memory 120. The graphics display information is read from graphics memory 120 at the appropriate time and properly transformed into graphics pixel data at graphics pixel data bus 142. Multiplexor 124 multiplexes the graphics pixel data 142 and the video pixel data 138 into pixel data signal 144. In this manner the multiplexor 124 "overlays" video pixel data on top of the graphics pixel data when the overlay window signal 148 is generated and output into the pixel data signal 144. The RAMDAC 126 converts the pixel data signal 144 into an analog RGB signal 146 for display on a graphics CRT monitor 129. The "overlay" technique requires a synchronization between the video pixel data 138 and graphics pixel data 140 and the generation of the overlay window signal 148 for the multiplexor 124. This requires that the source of video pixel data and the source of graphics pixel data have the dot clock (pixel clock), horizontal sync, and vertical sync signals all synchronized together. Control lines 139 connected to the digital video processor and VGA controller 128 accomplish the synchronization and control of the flow of video pixel data into the VGA controller. The most difficult synchronization is the dot clock (pixel clock) because the video pixel data is generated at a much different rate than that of the graphics pixel data. For example, graphics monitors operate at various dot clock frequencies such as 25 Mhz for a 640.times.480 (VGA) resolution, 40 Mhz dot clock for a 800.times.600, and 65 Mhz dot clock for 1024.times.768 resolution while the video signal 132 is generated at typically a 14 MHz rate. Furthermore assuming a 30 Hz video refresh rate and a 60 Hz graphics refresh rate, in approximately every 33 ms a new frame of video data is provided while every 15 ms a new frame of graphics data is provided. In order to accommodate the graphics monitor operation dot clock frequency, the video signal must be buffered and matched to the dot clock frequency. This is accomplished by the digital video processor by use of the video frame buffer memory 116. Thus video information is stored in the video frame buffer memory at a video rate and displayed onto the graphics monitor at a graphics frequency rate having synchronized horizontal and vertical sync signals for the appropriate dot clock frequency of the graphics monitor 129.
A disadvantage to the "overlay" method described above is that it requires two different memory arrays having separate address and data buses, a digital video processor, and a VGA controller. It is desirable to reduce the number of component parts and the necessary interconnect in order to provide a lower cost solution for combining a motion video window within a graphics display. What is needed is a different technique of providing a motion video window within a graphics window. It is further desirable to reduce the number of parts in order to reduce the power consumption and space utilization of components in order to provide a portable computer or laptop solution.
Another disadvantage to the "overlay" method is the required synchronization between dot clock, horizontal sync, and vertical sync signals for video pixel data and graphics pixel data. Thus it is desirable to eliminate the synchronization requirement and provide an alternate method of mixing the different pixel data types together.